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SOUS  EFFECTS  OF  VAEIATION  OF  CONTfiOL  LAE  PARAIUTBBS 
IN  THE  SIEULAfSD  BT  31  o  SYSTEM 


by 


LoS.  Speight 


SUMMARY 


In  the  ET  316  eyateni  the  operator  is  required  to  track 
targets  Tlsually  Tie  a  aerro-aaaisted  .sight.  It  eeems  olear 
that  the  choice  of  control  lav,  relating  the  operator's  Joystick 
deflection  to  the  trs'sker  output,  will  have  a  oonsldarable 
influence  on  tracking  error. 

An  experinent  hss  been  conducted  wherein  control  law 
parameters  have  been  veried  for  three  different  simulated  target 
oourses.  Two  of  the  targets  were  straight  and  level  fliers  but 
with  different  crossing  distanoes,  and  the  remaining  target  had 
an  added  2g  vertical  weave.  Eleven  PA  personnel  were  employed 
as  subjects . 

The  results  have  been  presented  as  a  series  of  approximate 
functional  relationships  between  tracking  aoouraoy  or  acquisition 
snd  settling  time  on  the  one  hand,  and  control  law  parameters  on 
the  other.  It  has  been  concluded  that; 


(a)  The  relationship  between  tracking  accuracy  or 
acquisition  and  settling  time  and  oontrol  law 
parameters  is  dependent  on  the  particular  target 
course,  and  so  there  is  unlikely  to  be  a  unique 
opti.ffluin  law  suitable  for  all  targets. 

(b)  For  the  targets  used  In  the  experiment,  a  control 
law  with  parameters  roughly  as  follows  appears  to 
be  a  reasonable  compromise: 


position 

velocity 

acceleration 


1.1  deg. 

4.8  dec/seo. 

O 

2.2  deg/aeo  . 
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SOME  EmCTS  OF  VASIATION  OF  CONTROL  UW  PARAMETERS 
IN  THE  SIMUUIED  ST  31 6  SYSTEM 


by 


L.R.  Speight 


INTRODUCTION 


BaokgrouBd 

1 .  In  the  ST  316  ayetem  the  operator  la  required  to  traolc 
targeta  Tlaually  via  a  aerro-aaalated  al|^t.  Quite  obTioualy 
the  oholoe  of  control  law,  relating  the  operator* a  Joyatlok 
deflection  to  the  traoker  output,  will  hare  a  oonaiderable 
influence  on  the  expected  tracking  error.  Aooordingly,  thla 
aapeot  of  deaign  haa  been  atudled  experlBentally  both  by  the 
Royal  Radar  SatabllahBent  (RRS),  Malvern,  and  by  the  Britiah 
Alroraft  Coz*poratlon  (BAC),  Stevenage,  although  the  llnea  of 
approach  uaed  by  theae  two  agenolea  have  been  rather  different. 

2.  In  a  preliminary  experiment^ ^ ^  RRB  oompared  a  pure  velocity 
control  with  an  aooeleration  control  (the  latter  with  a  email 
poaition  component  added),  ualng  aa  a  oritarion  of  tracking 
aoouraoy  the  integrated  root  mean  aquare  (r.m.a.)  error  over  a 
defined  portion  of  the  run.  Althou^  dlfferanoea  were  email,  the 
velocity  control  aeemed  to  be  ali^tly  auperior.  However,  on 
theoretioal  grounda  SearleC^}  had  argued  that  a  "mixed”  law  with 
poaition,  velocity  and  acceleration  terma  in  the  ratio  1:4:S 
ahould  be  optimal  for  tracking.  An  experiment  comparing  a  law  of 
thia  kind  with  velocity  controla  with  different  amowta  of  poaition 
aiding  confirmed  the  auperiority  of  the  "mixed”  lawC^).  Thla  aerlea 
of  Inveatlgationa  thua  led  to  a  preferred  law  auoh  that  full 
defleotion  of  the  Joyatlok  control  gave  a  demand  of: 


poaition  0.3  deg. 

velocity  2  de^aeo. 

2 

aooeleration  4  de^aeo  . 

and  thla  waa  uaed  by  ACRE  (now  AFRE)  Phm  they  inveatlgated  the 

effeota  of  different  typea  of  oontrol(7). 

3.  The  BAC  approach  waa  more  empirical.  A  number  of  velocity 
controla  with  different  amounta  of  poaition  aiding  were  examined 
to  aeleot  one  which  gave  good  reaulta  over  the  major  part  of  the 


» 
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missile  ooverage^^^ .  It  was  concluded  that  the  best  law  to  aohlewe 
this  aim  was  one  that  for  full  joystick  deflection  gave: 

position  1o43  dego  (0,025  rad) 

Toleoity  4. 30  deg/seo.  (0,075  rad/seo.) 

However,  this  law  oould  not  generate  rates  sufficient  to  track  some 
olasses  of  engageable  target,  A  satisfactory  means  of  producing 
this  additional  oapablllty,  without  prejudicing  performance  over 
the  main  portion  of  the  missile  coverage,  was  to  add  a  oertaln 
amount  of  acoeleration  control.  This  series  of  experiments  thus 
led  to  a  final  law  (as  given  in  Reference  3)  such  that  full  Joy- 
stiok  defleotion  gave: 

position  1.53  deg.  (0.026?  rad) 

velooity  4.58  deg/seo.  (0.06  rad/seo.) 

5  2 

acoeleration  2.29  deg/seo  .  (0.04  rad/seo  .) 

4.  The  present  experiment  was  devised  to  complement  these  previous 
investigations.  It  aimed  at  expressing  tracking  errors  as  an 
approximate  mathematloal  function  of  control  law  parameters,  at  least 
over  a  limited  range  of  these  latter  and  for  a  limited  number  of 
target  ooursea.  As  well  as  desorlblng  the  results  in  a  general  way, 
it  was  hoped  that  the  "response  surfaces"  so  fitted  to  the  data  night 
indicate  optimal  values  of  oontz<ol  law  oonstants.  It  was  also  planned 
to  investigate  the  related  problem  of  aoqulsitlon  and  settling  time  in 
similar  fashion  during  the  same  experiment. 


3>  To  examine  the  effect  of  variations  of  control  law  oonstants 
on  traoking  accuracy  and  on  acquisition  and  settling  time  in  the 
simulated  ET  31 6  system. 


METHOD 


fifSSr^ 

6.  In  the  type  of  control  law  considered  there  are  three  constants 
the  position,  velooity  and  acceleration  components  -  whioh  in  theory 
oan  be  varied  independently.  In  praotloe,  however,  there  are  some 
oombinatlons  of  velooity  and  acceleration  which  cannot  be  used  in 
oonjunotion:  if  they  are  together  too  high  in  value,  the  system  is 
too  sensitive  for  aoourate  traoking,  and  if  together  they  are  too 
low,  the  system  beoomes  incapable  of  following  those  targets  with 
the  fastest  rates.  This  suggests  that  for  the  practical  purposes  of 
this  investigation  these  two  parameters  can  be  combined  into  a 
single  "velooity  vs  acoeleration"  dimension,  at  one  extreme  of  whioh 
would  be  pure  velooity  control,  and  at  the  other  pure  acceleration 
oontrol,  with  different  admixtures  of  these  two  in  between.  In  the 
following  experiment  this  dimension  has  been  defined  by  the  line 

V  +  2A  =  10 

where  V  is  the  value  of  the  velooity  component  in  deg/seo. 

2 

and  A  is  the  value  of  the  acceleration  ooiq>onent  in  deg/seo  . 
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7.  -  In  the  general  experimental  method  uaed  it  le  aeeumed  that  a 
functional  relatlonahlp  exists  between  the  mapiltude  of  traoklng 
errors  and  control  law  parameters.  The  form  of  this  relationship 
Is  unknown,  but  It  Is  further  assumed  that  oTer  a  sufficiently 
small  range  It  can  be  approximated  by  a  polynomial  of  the  type 

2  2 

V  ®  o  '^1^1  '^2^2  ^  ^  ^22^2  ^  '^12^1^2 

where  V  Is  the  expected  magnitude  of  traoklng  error 

x^  la  the  value  of  the  position  component  expressed 
on  a  suitable  scale 

x^  Is  the  value  of  the  veloolty-aooeleratlon  component 
expressed  on  a  suitable  scale 

and  Q  t  eto.  are  the  lower-order  terms  In  a  Taylor  series 

expansion  of  the  "true"  function. 

Estimates  b^  ,  b^  ,  etc.  of  the  parameters  p  eto.  may  be 

obtained  by  experimentation,  using  one  of  a  class  of  expezlmental 
designs  which  have  been  evolved  for  this  purpose.  A  full  desorlption 
of  the  deslgp  uaed  In  this  experiment  Is  given  In  Appendix  B. 

Pilot  Experiments 

8.  As  a  lower-order  polynomial  Is  likely  to  be  an  adequate  model 
over  a  limited  range  only,  and  as  In  any  case  our  Interest  Is  oon- 
flned  to  that  region  where  traoklng  optima  are  likely  to  lie,  some 
ssiall  scale  pilot  experiments  were  carried  out  In  order  to: 

(a)  Determine  the  range  of  values  over  which  the 
main  experiment  should  be  oonduoted. 

(b)  Suggest  suitable  scales  in  terms  of  wfaloh  the 
control  law  parameters  should  be  expressed. 

(o)  Throw  some  light  on  other  effects,  suoh  as  transfer 
of  training,  day-to-day  variation,  learning  and 
wlthin-blook  trends,  wfaloh  mi^t  have  some  bearing 
on  the  conduct  and  design  of  the  main  experiment. 


Although  It  would  have  been  preferable  to  use  properly  representative 
subjeots  for  the  pilot  experiments,  the  supply  of  Army  personnel 
available  for  suoh  purposes  Is  strictly  limited,  and  two  soiontiflo 
staff  acted  In  this  oapaoity. 

9.  The  experiments  were  oonoemed  with  traoklng  aoouraoy  only 
and  are  desorlbed  briefly  In  Appendix  A,  and  the  l^lioatlons  they 
were  thought  to  have  for  the  design  of  the  main  experiment  are 
dlsoussed.  In  particular.  It  was  oonoluded  that  a  suitable  scale 
for  the  position  component  was  given  by  the  transformation 

where  P  is  the  value  of  the  position  component  In  degrees.  No 
simple  transform&tlon  for  the  velocity  vs  aooeleration  dimension 
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proved  adequate,  but  the  folloikng  rather  artlfloial  soale  provided 
a  good  fit  to  the  data: 

Xj  =  V(V  +  0.5A),  (V  +  2A  =  10) 

where  V  is  the  value  of  the  velooity  oonponent  in  dog/aeo,  and 
A  is  the  value  of  the  aooeleration  oonponent  in  de|/aeo^.  The 
experimental  variables  were  therefore  defined  in  this  way, 

Subjeots 

10.  Eleven  personnel  from  37  A.D.  Regt  RA  took  part  in  this 
experiment.  This  was  the  sane  sample  of  12  (less  one  man  un¬ 
avoidably  absent  on  leave)  who  had  partioipated  in  a  study  of 
skill  retention  reported  in  Referenoe  8.  It  was  shown  there  that 
these  12  oould  be  regarded  as  a  representative  sample  from  the 
potential  user  population. 

1 1 .  Prior  to  the  present  investigation,  these  operators  had 
undertaken  a  minimum  of  1 28  traoking  runs,  using  for  the  most 
part  a  oontrol  law  with  the  following  oonstanta: 

position  1.3  deg. 

velooity  4.6  de^seo. 

2 

aooeleration  2.4  de^seo 

Equipment 

12.  The  experiment  was  oonduoted  at  RRB,  Malvern,  using  the 
simulation  equipment  developed  there.  RRI  have  provided  a  fairly 
full  desorlption  of  this  equipment  In  Appendix  A  of  Referenoe  ?• 

The  operator's  station  oonslsted  of  a  padded  ohair  in  front  of  an 
elbow-level,  wooden  desk-top,  at  the  far  and  of  whiob  was  the 
display,  its  oentre  raised  13^"  above  the  surfaoe.  This  display 
oonslsted  of  a  14"  EMI  Marie  IT,  type  Tni6/litA,  TT  monitor, 
representing  a  4*8^  by  3*8°  field  of  view.  It  was  situated 
rou^ily  22"  from  the  operator's  eyes,  althou^  this  varied  with 
posture,  whioh  was  uncontrolled.  Eleotronloally  generated  oross- 
wires  were  used,  with  a  small  gap  at  their  Junction.  Both  target 
and  oross-wires  were  dark  against  a  bri^t  background.  The  oontrol 
was  a  miniature  ft>ee-movlng  Joystick,  let  into  the  desk-top.  It 
measured  3/18"  from  pivot  to  tip  and  permitted  an  angular  movement 
of  ±  32^°.  The  sense  of  this  oontrol  was  suoh  that  defleotlon  of 
the  Joystiok  to  the  left  moved  the  target  spot  to  the  left,  and 

a  defleotlon  towards  the  operator's  body  moved  the  target  spot 
downwards. 

Proeedure 

13«  The  targets  used  In  this  experiment  were  all  of  velooity 
1,000  ft/seo.  and  initial  slant  range  30,000  ft.  Three  target 
oourses  were  used,  with  orossing  distanoes  of: 

(a)  2,300  ft. 

(b)  5,000  ft. 

(o)  10,000  ft. 
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The  first  and  last  of  these  were  straight  and  level  fliers  with 
height  zero  feet,  hut  course  (b)  had  an  added  2g  vertical  weave 
(sine  wave)  with  a  period  of  12  sec.  and  peak  amplitude  234-  ft. 

lif.  At  the  start  of  each  run  the  target  was  displaced  from  the 
cro3S"Wirea  200  ft.  vertiocdly  and  200  ft.  to  the  ri^t.  It  was 
hoped  in  this  way  to  obtain  a  rou^  estimate  of  the  effect  of  the 
control,  law  on  target  acquisition  and  settling  time.  Each  tracking 
run  lasted  27  sec,  tracking  errors  being  Integrated  over  the  last 
two-thirds  of  this  period.  The  index  used  for  this  purpose  was 
r.m.s.  radial  error  measured  in  feet  (although  the  analysis  which 
follows  is  conducted  in  terms  of  log  r.m.s.  error,  as  this  trans¬ 
formation  is  required  for  statistical  validity).  Pen  records  of 
tracking  errors  were  taken  over  the  first  9  seo.  of  each  run  and 
these  showed  the  miss  distance,  in  feet,  in  both  vertical  and 
horizontal  planes.  For  the  purposes  of  this  experiment  "acquisition 
and  settling  time"  was  defined  as  the  time  from  the  start  of  the 
run  until  the  operator  reduced  his  error  in  both  planes  to  less 
than  20  ft,  subsequently  holding  his  error  within  these  boundaries 
for  at  least  2  sec.  Acquisition  time  was  thus  detez*mlned  in  each 
case  by  measurement  on  the  appropriate  pen  record. 

13.  Each  of  the  11  operators  undertook  11  blocks  of  tracking 
runs,  each  block  being  with  a  different  control  law.  The  control 
law  parameters  used  in  this  experiment  are  listed  below: 


TABU  I 


Block 

Number 

(Control  Law) 

Control  Law  Parameters 

Position 

Velocity 

Acceleration 

Transformed  Scale 

(deg/seo) 

(deg/seo^) 

*1 

X2 

.5  ) 

1.8 

4.4 

2.8 

0.25527 

0.48276 

4 

1.2 

5.2 

2.4 

0.07918 

0.37500 

5 

1.2 

3.737 

3.131 

0.07918 

0.59052 

6 

2.7 

5.2 

2.4 

0.43136 

0.37500 

7 

2.7 

3.737 

3.131 

0.43136 

0.59052 

8 

1.015 

4.4 

2.8 

0.00625 

0.48276 

9 

,5.194 

4.4 

2.6 

0.50430 

0.48276 

10 

1.8 

5.582 

2.209 

0.25527 

0.33037 

11 

1.8 

3.495 

3.253 

0.25527 

0.63515 

These  blocks  were  arranged  in  a  Latin  Square  design  to  control  any 
long-term  trends,  such  as  learning  effects.  Eaoh  block  consisted  of 
7  runs,  the  first  three  being  of  target  course  (o),  the  next  two  of 
course  (a),  and  the  remaining  two  of  course  (b),  defined  in  Paragraph 
13.  The  scores  from  the  first  run  in  each  block  were  excluded  from 
the  analysis,  l.n  order  to  minimise  short-term  transfer  effects.  The 
scored  pair  of  runs  for  eaoh  target  course  consisted  of  a  left  and  a 
right  crossing  tai'get,  presented  in  random  order. 
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fiSSULTS 


16.  Th«  statistical  analysis  of  the  data  Is  given  In  Appendix  B. 
The  analysis  yielded  fitted  equations  of  the  following  type: 


y  *  ^>0  *  "  ^2^2  ♦  *  '>22*2^  *  ^2*1*2 

where 

y  Is  the  estimated  response  (log  r.m.s.  error  for 
traoklng  aoouraoy  or  time  In  seconds  for 
acquisition  and  settling) 


(2) 


x^  Is  the  position  component  on  the  transformed  scale 

x^  Is  the  velocity  vs  acceleration  component  on  the 
transformed  scale 


and  b  (b . ,  eto.  are  estimates  of  the  constants  etc. 

®  ’  In  Equation  (1).  ®  ^ 


It  was  found  that  the  data  demanded  separate  equations  for  eaoh 
target  course.  The  values  of  the  fitted  constants  are  given  In 
Table  II. 


TABIE  II 


Target  Course 

Fitted  Constanta 

(Crossing  Distance) 

■a 

"2 

Nl 

^22 

^2 

Traoklng  Aoouraoy 

10,000  ft 

2,500  ft 

5,000  ft  4  weave 

0.4815 

0.6057 

0.9224 

0.257D 

-0.3579 

-0.1109 

0.2990 

0.5655 

-0.1247 

0.6952 

0.5439 

0.7774 

0.1081 

-0.7377 

-0.0261 

0  0 

Acquisition  Tims 

10,000  ft 

2,500  ft 

5,000  ft  4  weave 

0.9613 

0.8040 

1.1579 

-6.0827 

-0.6023 

-7.4643 

4.9437 

3.0972 

3.9556 

12.1626 

5.5201 

5.4423 

-2.2141 

-O.8197 

-3.8654 

7.3660 

-2.8446 

16.2292 

17«  By  mere  Inspeotlon  of  the  numerloal  values  of  these  sets  of 
ooefflolents  It  Is  difficult  to  visualise  the  surfaces  which  they 
represent.  They  can  be  Illustrated  graphically,  however,  by  drawing 
a  system  of  contours  of  equal  response  In  the  space  of  x^  and  X2. 

In  this  way  a  kind  of  map  of  traoklng  aoouraoy  or  acquisition  time 
can  be  built  up,  so  that  the  way  In  which  estimated  response  varies 
with  the  values  of  control  law  constants  may  readily  be  appreciated. 
Such  naps  are  given  In  Figures  1-6.  The  points  at  vhloh  msasurenents 
were  taken  are  Indicated  In  these  Figures,  together  with  the  estlmaftes 
of  response  actually  obtained  at  these  points.  Also  shown.  In  the 
form  of  a  long  dotted  line.  Is  the  major  axis  of  the  fitted  system 
(l.e.  the  line  along  which  there  Is  J.east  variation  of  response). 

In  those  oases  irtiere  there  Is  a  stationary  point  within  the  area 
covered  by  the  experiment,  the  minor  axis  Is  also  Indicated  (althou^ 
In  Appendix  B,  Paragraph  B.7,  e  note  of  caution  Is  sounded  on  the 
Interpretation  of  those  principal  axes).  For  all  three  target  courses 
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Figure  I 


Track i ng  accuracy , 
distance  10,000  ft. 
spaced  at  Intervals 
r.m.s.  error. 


Target  crossing 
Contours  are 
of  0.02  log 


Figure  2 


Tracking  accuracy, 
distance  2,500  ft. 
spaced  at  interval 
r.m.s.  error. 


Target  crossing 
Contours  are 
s  of  0.02  log 
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Figure  3 

Tracking  accuracy.  Target  crossing 
distance  5,000  ft.  with  2g  vertic^ 
weave.  Contours  are  spaced  at 
Intervals  of  0.02  log  r.m.s.  error. 
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both  for  tracking  aoouraoy  and  aoqvalsltlon  tine,  ourrature  along 
tha  aajor  axis  falls  far  short  of  statistical  significance.  Thus 
the  data  are  not  Inconsistent  with  the  Interpretation  that  they 
reflect  valley  (or  Inverted  ridge)  systeoks,  with  all  curvature 
taking  place  in  one  dimension. 

18.  It  will  be  understood  that,  unless  the  slse  of  the  experiment 
Is  extremely  large,  response  surfaces  will  be  determined  with 
appreciable  experimental  error.  It  would  therefore  be  unwise  to 
place  too  great  a  rellanoe  on  the  finer  details  of  the  Figures 
shown  here.  This  may  be  seen  by  oonsldarlng  the  width  of  confidence 
Intervals  for  estimates  of  response.  Although  these  will  vary  for 
different  locations  In  the  experimental  region,  all  points 
equidistant  from  the  centre  of  the  present  design  will  have  intervals 
of  equal  slse.  If,  for  purposes  of  Illustration,  we  axhltrarlly 
ohooae  any  spot  on  the  outer  olroumferenoe  of  experimental  points, 
then  the  width  of  3P^  and  90^  oonfldenoe  Intervals  at  this  location 
will  be  as  given  In  Table  III. 


TABU  III 


Target  Course 
(Crossing  Dlstanoe) 

Width  of 

Confidence  Interval 

Unit  of 

90JJ 

Tracking  Aoouraoy 

10,000  ft 

2,900  ft 

9,000  ft  weave 

000 

e  e  e 

000 

0.0901 

0.01,93 

0.0372 

log  r.m.s. 
error  (ft) 

Acquisition  Time 

10,000  ft 

2,900  ft 

9,000  ft  •»-  weave 

0.207 

0.182 

0.296 

0.907 

0.447 

0.729 

time  (see) 

19.  There  Is  a  strong  positive  relationship  between  eaoh 
operator's  mean  tracking  score  and  his  mean  aoqulsitlon  time, 
taken  over  all  control  laws  and  all  target  oouraes  in  this 
experiment.  The  correlation  between  these  two  indloea  was  0.68, 
whloh  attains  a  high  level  of  atatlstloal  slgnifioanoe.  On  the 
other  hand,  the  relationship  between  IntelUgenoe  test  results 
(TSO  soore)  and  tracking  perfomanoe  was  small  and  very  fhr 
from  attaining  statistical  sl^ilfloanoo. 

20.  From  the  data  of  this  expu'lment  the  population  standard 
deviation  for  tracking  error  has  been  estimated  as  0.0636  unita. 

If  we  oan  assume  that  individual  tracking  abilities  approxinately 
follow  a  log  normal  distribution,  this  ijqplies  that  the  dlfferenoe 
between  operators  at  tha  9th  and  99th  peroentilea  is  a  factor  of 
1.9«  There  was  no  evldenoe  to  suggest  that  this  factor  differed 
for  different  target  courses. 


DISCUSSION 


21.  Before  oonsiderlng  the  results  obtained  in  this  experiment, 
it  would  perhaps  be  as  well  to  re-emphaaise  the  dlfferenoe  between 
BAG'S  approach  to  control  law  optimisation  and  that  adopted  here. 
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It  will  be  reoalled  thet  BAC  attempted  to  ensure  that  eiror  would 
hot  be  excessive  at  any  point  in  the  missile's  ooverage;  whereas 
we  have  attempted  to  minimize  the  average  eiror  during  an  engagement, 
giving  full  weighting  to  all  segments  of  the  target  run  within 
defined  limits.  Roughly  speaking,  the  former  strategy  would  be 
moot  appropiriate  if  it  were  assumed  that  missile  interception 
would  tend  to  cluster  fairly  closely  around  one  range  point  in 
future  engagements,  the  location  of  this  point  being  unknown  at 
present.  The  latter  strategy  would  be  more  appropriate  if  one 
could  assume  that  the  probability  of  missile  interception  would 
be  fairly  evenly  distributed  over  a  wide  range  band.  In  fact,  our 
present  state  of  knowledge  probably  lies  somewhere  between  these 
two  extremes  (although  possibly  nearer  the  former  than  the  latter) 
and  it  is  this  reasoning  which  has  led  us  to  believe  that  the  two 
approaches  outlined  are  oomplementary.  It  is  worth  pointing  out, 
however,  that  the  results  obtained  by  the  two  optimization  methods 
will  not  necessarily  be  identical.  Of  course,  if  each  method  leads 
to  similar  conclusions,  then  the  confidence  in  one's  final  choice 
of  control  law  would  obviously  be  that  much  the  greater. 

22.  From  Annex  A  it  will  be  seen  that  the  final  pilot  experiment 
showed  a  clear-out  optimum  for  the  non-manoeuvring,  10,000  ft. 
crossing  distance  target,  and  an  Inverted  ridge  system  for  a  2,300  ft 
crossing  distance  target  with  3g  vertical  weave.  Both  results  were 
misleading  as  a  guide  to  the  findings  of  the  main  experiment,  as 

no  true  optima  were  found  for  straight  and  level  fliers  in  the 
latter  oase,  and  for  the  manoeuvring  target  the  inverted  ridge 
was  located  differently.  One  factor  which  might  be  thought  to 
influence  results  i^s  that  the  expeidmental  sample  consisted  of 
two  groups  with  different  early  training  histories.  Before 
working  with  the  BAC  control  law  (l .5A-.6/2.4) .  four  of  the 
sample  had  trained  on  the  RRE  law  (0.5/2.0/4.0) ,  whereas  the 
remaining  seven  had  trained  on  the  BAC  law  from  the  outset. 

It  is  possible,  then,  that  early  training  might  have  had  a  long¬ 
term  residual  effect,  and  that  the  results,  in  fact,  spring  from 
two  separate  groups  with  different  optima.  As  there  was  no  marked 
learning  effect,  it  was  possible  to  analyse  these  two  sub-samples 
separately.  The  two  sets  of  response  surfaces  so  generated  proved 
very  similar  to  each  other  and  to  the  set  based  on  the  pooled 
data,  and  so  we  have  discounted  early  training  as  a  possible 
explanation.  It  is  felt,  however,  that  the  dlsorepanoiea  between 
the  results  of  the  pilot  experiment  and  those  of  the  main 
experiment  do  demonstrate  the  importance  of  correct  sampling. 

The  fact  that  the  most  noticeable  difference  between  samples 
drawn  from  different  populations  is  often  one  of  general 
Intelligence  or  educational  attainment  should  not  be  allowed  to 
obscure  the  possibility  of  differences  in  less  obvious,  less 
easily  quantified,  but  still  important  associated  variables  (suoh 
as  opportunities  for  exercising  other  control  skilla  -  oar  driving, 
for  instance  -  or  even  suoh  manipulative  skills  as  drawing  or 
writing) . 

23.  Over  the  range  of  values  covered  in  the  experiment,  variations 
in  control  law  parameters  affected  tracking  accuracy  relatively 
little  for  the  2,500  ft.  crossing  distance,  rather  more  for  the 
10,000  ft;  target,  but  for  the  weaving  target  the  effect  was  quite 
marked.  If  results  had  been  quoted  in  terms  of  r.m.s.  errors 
rather  than  log  r.m.s.  errors,  this  difference  would,  of  course, 
have  appeared  even  greater.  If  aircraft  are  as  likely  to  manoeuvre 
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as  to  fly  straight  and  leval,  then  thsss  results  suggest  that 
greater  wei.ght  should  be  given  to  the  former  class  of  target 
than  to  the  latter  in  optimizing  the  control  lawo  As  for 
acquisition  time,  as  might  be  expected,  control  law  variations 
have  the  greatest  effect  on  targets  with  the  greatest  initial 
rates  (i,o.  the  weaving  target,  and  that  with  a  crossing  distance 
of  10,000  ft). 


24-.  The  major  axes  of  the  six  response  surfaces  obtained 
(three  for  tracking  accuracy  and  three  for  acquisition  time) 
are  reproduced  in  Figure  7.  It  will  be  seen  that  they  come 
quite  close  together  in  the  bottom  left-hand  corner  of  the 
diagram,  so  that  a  control  law  represented  by  the  experimental 
point  ringed  should  bo  reasonably  satisfactory  for  all  three 
targets  used  in  this  experiment.  Irdeed,  if  the  results  for  all 


Figure  7:  Major  Axes  of  Response  Sxirfaoes.  Target  Courses: 
A  -  10,000  ft  orossing  distance 

B  -  3,000  ft  orossing  distance 

C  -  2,300  ft  orossing  distance  with  2g 

vertical  weave 
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three  target  oourses  are  pooled,  the  responae  aurfaoea  depleted 
in  Figures  8  and  9  result,  and  consideration  of  these  latter  leads 
to  substantially  the  same  choice  of  control  law.  (it  Is  unfortunate, 
howerer,  that  the  choice  of  experimental  points  prompted  by  the 
pilot  Investigation  should  have  proved  to  be  so  badly  centred  for 
this  sample.)  The  point  Indicated  represents  a  control  law  with 
the  following  parameters: 

position  1.2  deg. 

velocity  5.2  de^seo. 

2 

acceleration  2.4  dag/seo  . 

Scaled  down  to  the  sensitivity  of  the  BAG  law  (which  latter  satisfies 
the  condition  V  •«-  2A  =  9  >2),  the  values  of  the  parameters  would  read: 


position 


1 .1  deg. 


velocity  '4«78  d®g/seo. 

\  .  r.  r,  2 

acceleration  .2.21  deg/seo  . 

i  t 

Apart  from  the  rather  smaller  position  term  (required  for  good 
performance  against  a  manoeuvring  t^get) ,  this  is  very  similar  to 
the  law  proposed  by  BAG  (l .33/4«5S/k.29).  However,  even  discounting 
the  approximate  nature  of  this  "optimum".  It  is  obvious  that  k  fallr 
amount  of  latitude  la  possible  In  the  neighbourhood  of  this  point 
without  seriously  affecting  tracking  accuracy  or  acquisition  ilme. 
There  Is,  In  fact,  a  hint  In  the  data  that  a  control  law  with  an 
even  hl^er  ratio  of  velocity  to  acceleration  tenis  would  not 
degi<ade  t^aokltig  performanoe  unduly  and  might  ali  aoqulsltlonj 
but  such  a  law  would  lie  outside  thO  area  covered  by  the  experiment, 
where  the  width  of  confidence  Interval  would  be  quite  large. 


25 •  Turning  to  the  subsidiary  results,  the  strong  positive 
relationship  between  acquisition  performanoe  and  tracking  accuracy 
Is  encouraging.  It  has  been  found  with  Vlgllant(3)  that  performance 
against  short-range  targets  (which  Is  considered  to  be  mainly  an 
acquisition  task)  Is  negatively  correlated- with  performanoe  against 
long-range  targets  (which  Is  mainly  a  tracking  task);  If  this 
result  had  been  repeated  with  ET  316,  It  would  have  been  most 
Inconvenient. 

26,  In  early  experiments  with  simulations  of  ET  316,  a  strong 
positive  relationship  was  found  between  Intelllgenoe  and  tracking 
accuracy.  A  previous  reports)  suggested  that  It  might  be  unwise 
to  accept  these  results  too  unorltloally,  as  it  was  felt  that  this 
relationship  might  disappear  with. verbal  guidance  during  early 
learning,  and  with  prolonged  training.  These  latter  conditions 
apply  to  the  present  group  of  operators,  and  the  correlation  has 
Indeed  dropped  to  near-sero. 

27*  Finally,  It  should  be  pointed  out  that  the  estimate  of 
population  standard  deviation  (0.0833  log  units)  Is  obtained  on 
the  most  highly- trained  sample  likely  to  be  encountered  prior  to 
evaluation. 
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Figure  8 


T  rack i ng 
spaced 


accuracy.  All  three  target  courses  pooled, 
at  intervals  of  0.02  log  r.m.s.  error. 


Contours  are 
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Figure  9  Acquisition  and  settling  time.  All  three  target  courses  pooled. 
Contours  are  spaced  at  Intervals  of  secs. 
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SUMUAEY  JkND  CONCLUSIONS 


28.  An  experiment  has  been  oonduoted  to  etudy  some  effects  of 
Tsrlatlons  of  control  law  parameters  In  the  simulated  ET  316  system. 
The  results  have  bean  presented  as  a  series  of  approximate 
functional  relationships  between  tracking  accuracy  or  acquisition 
and  settling  time  and  control  law  parameters.  It  has  been 
concluded  that: 

(a)  The  relationship  between  tracking  accuracy  or 
acquisition  and  settling  time  and  control  law 
parameters  is  dependant  on  the  particular 
target  course,  and  so  there  is  unlikely  to  be 
a  unique  optimum  law  suitable  for  all  targets. 

(b)  For  the  targets  used  in  this  experiment  a 
control  law  with  parameters  roughly  as 
follows  appears  to  be  a  reasonable 
oompromlae: 

position  1.1  deg. 

velocity  4>8  deg/seo. 

2 

acceleration  2.2  deg/seo  . 


ACXNOWLBDGBISNTS 


This  experiment  was  oonduoted  at  RSB,  Ualvem,  on  the 
equipment  developed  there,  and  onoe  more  the  author  wishes  to 
express  his  thanks,  in  paz^lcular  to  Messrs.  J.S.  Blokerdlke, 
Gr.A.  Hawkins  and  J.H.  Hewlett  .for  their  help  and  encouragement. 

Grateful  acknowledgement  is  also  made  of  the  efforts  of 
personnel  from  37  AD  Regt  RA  idio  participated  in  the  experiment, 
and  of  WO  II  Soovell  of  the  School  of  ArtlUeiT’,  Manorbler,  who 
greatly  assisted  in  its  administration. 


15 

CONFIDENTIAL 


CON?IDENnAL 


SBFSSSNCXS 


1.  Blokerdlke,  J.S.  Preliminary  reaults  on  optical  tracking 
accuracy.  HUE,  Malvern.  WB  Memo.  No.  19^2 

(dONPIDENTIAL) . 


2.  Hewlett,  J.H.  Inveatlgatlon  of  the  effect  on  traoklng 

performance  of  aeveral  alternative  types  of  tracker 
ccntrcl  law.  RRE,  Malvern.  WI  Dlv.  Memo.  No.  6/jHH. 
1963  (CONFIDENTIAL). 

3.  Humphrey,  P.B.  and  Brown,  T.B.  Selection  of  operators 

for  a  velocity  ccntrcUed  anti-tank  missile  system. 
AOBE  Repcrt  No.  13/63.  1963  (CONFIDENTIAL). 

it-.  MaoLusky,  O.J.R.  and  Thomas,  D.M.  ET  316  Project  Study, 
Appendix  A,  Target  Traoklng.  BAG,  Stevenage 
SSI/5075.  1963  (SECRET). 

5.  Rook,  P.E.  and  Thomas,  D.M.  Perforsunoe  of  sample  of 

Royal  Artillery  operators  on  simulated  manual 
traoklng  of  aircraft  targets.  BAG,  Stevenage 
SSI/5082.  1963  (SECRET). 

6.  Searla,  L.V.  Psyohologloal  studies  of  traoklng  behaviour. 

Part  IV.  The  Intermlttenoy  hypothesis  as  a  basis  for 
predicting  optimum  aiding  time  constant. 

NRL  Report  3872.  1951. 

7.  Spoilt,  L.R.  The  effect  of  type  of  oontrol  on  traoklng 

accuracy  In  the  simulated  ET  316  system. 

AORE  Report  No.  l/(;5.  1965  (SECRET). 

8.  Speight,  L.R.  Retention  of  traoklng  skill  and  "warm-up" 

effects  In  the  simulated  ET  316  system. 

APHE  Report  No.  2/65.  1965  (CONFIDENTIAL). 


16 

CONFIDENTIAL 


CONPIDENTIAl 


Aimxx  A 

PILOT  bzpebuients 


Swral 

A.1.  The  poflslhle  range  of  variations  of  the  three  control  law 
parameters  la,  of  course,  extremely  wide.  To  oarxy  out  a  grid  of 
experiments  sufficiently  widespread  to  stretch  over  the  whole  region 
of  possible  operation  would  be  an  undertaking  too  prodigal  to 
oontemplate.  It  is  therefore  necessary  to  restrict  our  interest 
to  a  region  in  iriilch  we  hope  tracking  optima  will  lie  and  so,  as 
indicated  in  Paragraph  8  of  the  main  text,  a  series  of  experiments 
was  planned  in  order  to: 

(a)  Determine  the  range  of  values  over  irtiloh  the  main 
experiment  should  be  conducted. 

(b)  Suggest  suitable  scales  in  terms  of  which  the 
control  law  parameters  should  be  expressed. 

(o)  Throw  llj^t  on  other  effects,  such  as  transfer 
of  training,  learning,  etc. 

A. 2.  Probably  the  most  elegant  and  economical  experimental  technique 
for  approaching  the  region  of  an  optimum  is  the  method  of  "steepest 
ascent"  pioneered  by  Box  and  others  (see,  for  example,  Reference  A.1). 
This  is  essentially  a  sequential  procedure,  and  its  effective  use 
requires  analysis  between  the  successive  experimental  steps.  Such  an 
arrangement  would  have  been  Impraotloal  in  the  present  case,  and  so 
the  classic  "one-faotor-at-a-time"  approach  has  been  used.  Two 
experiments  were  oonduoted  varying  the  velocity  and  acceleration 
terms  (regarding  these  as  one  factor),  holding  the  position  term 
constant.  This  was  followed  by  one  experiment  holding  velocity  and 
acceleration  terms  constant  and  varying  the  position  term.  The  fliwl 
experiment  was  conducted  with  a  grid  of  points  spread  out  in  both 
dimensions.  Detailed  analysis  is  omitted  in  the  following  paragraphs, 
but  the  experiments  and  the  results  obtained  are  briefly  described. 

Subjects 

A. 3.  Two  members  of  AFRE  Scientific  Staff  took  part  in  Experiments 
1  to  3.  One  of  these  subjects  was  omitted  for  Experiment  A. 

Prooedure 

A. 4.  In  terms  of  equipment,  and  in  general  details  of  target 
properties,  the  prooedure  used  in  these  experiments  were  similar  to 
those  in  the  main  experiment  (see  Paragraphs  12-14  of  main  text). 

However,  errors  were  Integnted  over  the  last  22  sec  of  each  run, 
allowing  only  3  seo  for  acquisition  and  settling. 

Experiments  1  and  2 

A. 3.  For  both  these  experiments  the  value  of  the  position  term  was 
held  constant  at  1.6  deg.  Measurements  were  then  taken  at  the  following 
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combinations 

of  velocity  and  acceleration  terms: 

(a) 

8. 4:0.8 

(b) 

6.8:1 ,6 

(c) 

5. 2:2. 4 

(d) 

3. 6:3. 2 

(e) 

2. 0:4.0 

2 

(do§/seo:do§/soc  ).  The  crossing  distances  for  the  three  target 
courses  were: 

(i) 

2,500  ft 

(ii) 

10,000  ft 

(iii) 

2,500  ft,  but  with  an  added  3  g  vertical  weave. 

A46.  In  Experiment  1  each  lubjeot  undertook  9  blocks  of  15  run* 
eaoh.  Each  block  was  devoted  to  one  target  course,  the  three 
courses  being  repeated  in  the  order  (i),  (ii),  (iii)  three  times. 

In  each  block  the  five  control  laws  were  taken  in  random  order, 
thz^e  targets  being  tracked  with  eaoh.  The  first  of  these  three 
runs  was  not  included  in  the  analysis.  However,  Inspeotlon  of  the 
results  showed  that,  in  general,  the  dlsoarded  rune  differed  very 
little  from  the  retained  runs,  and  that  there  was  no  apparent 
tendency  for  performance  to  improve  from  the  second  to  the  third 
run. 


A. 7.  In  Experiment  2  eaoh  subject  undertook  5  blocks  of  27  runs 
each.  Each  block  was  devoted  to  one  of  the  five  control  laws, 
those  being  arranged  in  a  different  random  order  for  each  subject. 
Throe  introductory  nms  (dlsoarded  for  the  purposes  of  analysis) 
were  given  at  the  start  of  eaoh  block,  followed  by  ei{d>t  runs  of 
each  of  the  three  target  courses,  these  24  being  undertaken  in 
random  order.  There  was  no  apparent  tendency  for  accuracy  to 
improve  within  any  one  block,  althou^  performance  did  seem  to 
become  more  variable  towards  the  end  of  the  block. 

A. 8,  It  was  evident  that,  if  the  experimental  points  chosen 
were  regarded  as  equidistant  on  a  velocity  vs  acceleration  scale, 
a  quadratic  relationship  would  be  quite  inadequate  to  descidbe  the 
data.  The  following  scales  were  therefore  devised: 

(a)  x^  «  A/V 

(b)  x^  =  A/(V  +  0.5A) 
and  (o)  Xg  =  A/(V  +  A) 

Using  these  scales,  quadratlo  relationships  were  fitted  to  the 
data,  taking  each  subject,  experiment  and  target  course  separately. 
Scale  (a)  gave  the  best  fit  Jji  five  cases;  scale  (b)  in  two;  and 
scale  (0)  in  five.  Taken. overall,  scale  (b)  seemed  to  give  the  best 
results,  as  even  when  it  was  intermediate  between  the  other  two, 
its  fit  was  little  Inferiot  to'  thb;  be*t. 
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A. 9.  The  straight  flying  targets  resembled  each  other  In  that 
their  curvature  was  quite  pronounced,  and  that  their  optima 
appeared  to,  be  similarly  placed,  7or  target  course  (i)  csdoulated 
optimal  acceleration  terms  were  3.00,  3.00,  2.96  and  2.86 
deg/seo^;  the  figures  for  target  course  (li)  being  2,20,  2.86, 

2.70  and  2.83  deg7sec^.  Averaging  over  these  two  target  courses 
thus  srlelds  an  optimum  at; 

velocity  4.40  deg^seo 

2 

acceleration  2.80  def/seo 

A. 10.  For  the  weaving  target  the  curvature  In  this  dimension 
was  far  less  pronounced.  Even  so,  the  data  yielded  remarkably 
consistent  optima,  the  rele^nt  acceleration  terms  being  3.27, 

3.29,  3.28  and  3.22  deg^seo^.  Averaging  once  again,  we  have  an 
optlmlm  at: 

velocity  3.46  deg/seo 

2 

acceleration  3.27  dej/seo 


Imperiment  3 

A.11.  For  this  experiment  the  velocity  and  acceleration  terms 
were  held  constant  at  4.4deg/seo.  and  2.8  deg/seo^,  respectively. 
The  following  values  of  position  elosent  were  used; 

(•)  0.5 

(b)  ■  1.0 

(0)  1.5 

(d)  2.0 

(e)  2.5  deg. 

Target  courses  were  the  same  as  most  In  Experiments  1  and  2. 

A. 12.  Each  subject  undertook  6  blocks  of  15  runs  each,  each 
block  being  devoted  to  one  target  course.  In  similar  fashion  to 
Experiment  1 .  Each  subject  and  each  target  course  was  analysed 
separately,  If  the  position  term  was  expressed  on  the  scale 
x^  s  log  P,  an  adequate  fit  was  obtained  by  a  quadratic  equation 
in  all  but  one  case.  The  optima  were  rather  scattered,  however, 
their  values  being  1.38,  1,80,  (2.80),  1.77,  2.00  and  1.26  deg. 

(the  bracketed  value  being  that  yielded  by  the  case  with  inadequate 
fit).  The  mean  of  these  values  Is  I.84  deg.  (I.64  deg,  if  the 
questionable  value  is  omitted). 

Experiment  4 

A. 13  The  final  experiment  of  this  series  employed  three  levels 
of  position  (0,8,  1.6  and  3.2  deg.)  and  four  levels  of  veloolty- 
aooeleration  (5. 2:2. 4,  4.4:2. 8,  3. 6:3.2  and  2.8:3. 6  (de^seo.  : 
de^seo^).  The  two  target  courses  had  crossing  distances  of: 

(i)  10,000  ft 

(11)  2,500  ft,  with  an  added  3g  vertical  weave. 
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Each  block  of  24  runs  was  confined  to  one  target  course,  and 
consisted  of  two  tracking  runs  for  each  of  the  12  control  laws, 
undertaken  in.  random  order „  Fire  blocks  were  undertaken  with 
target  course  (i)  and  four  blocks  with  target  course  (ii). 

A. 14.  The  two  target  courses  were  separated  for  the  purposes 
of  analysis.  Three  separate  analyses  were  carried  out  for  each 
target  course,  defining  the  position  scale  always  as  =  log  P, 
but  using  the  three  al.ternati’fre  definitions  for  the  velooity- 
acceleration  dimension: 

(a)  x,  =  A/V 

(b)  Xj,  =  A/(¥  +  0.5A) 

(e)  Xg  =  A/(V  +  A) 

The  scale  as  defined  by  (b)  gave  the  best  fit  for  target 
course  (i)  and  as  defined  by  (a)  for  target  course  (ii), 
althou^i  the  residual  sum  of  squares  varied  but  little  for 
different  scales  within  each  target  course,  and  all  were  well 
within  the  limits  of  experimental  error.  Most  important,  for 
target  course  (i)  the  three  scales  y5.elded  very  similar  optima 
as  follows: 


Velooity-A-oeeleration 

Pitted  Optima 

Scale  Defin.ed  by 

Position 

Velocity 

Acceleration 

(a)  X2  =  A/V 

ieg. 

1.86 

deg/sec. 

4.64 

deg^'sec^ 

2.68 

(b)  X2  =  A/(V  +  0.5A) 

1.84 

4,60 

2.70 

(c)  X2  A/(V  +  a) 

1.83 

4.62 

_ 

2.69 

A. 15.  Target  course  (iij  appeared  to  result  in  a  ridge  system, 
with  no  true  optimum.  Definition  (b)  was  therefore  accepted  for 
the  velocity-acoeleration  dimension.  With  x^  and  x^  thus  defined, 
the  final  equations  were: 

for  target  course  (i): 

^  =  0.7943-0. 9787s;.,  -  1.51653:2  +  0.8l56x^^  +  1.3176x2^  +  1.2062x^ 

V  'V 

and  for  tai’get  course  (ii); 

y  =  0.71 67-0. 1863X,  +  0.01253:2  +  0,6227x^^  +  O.OOISXg^  -  0.238QX., 

These  relationships  are  illustrated  in  Pi.gures  A.1  and  A. 2,  irtiere 
the  experimental  points,  together  with  the  measured  mean  log 
r.m.s.  errors,  are  also  shown. 
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0‘4  0-S  0-4  07 


(viocity  1^4.  »ce» ! 0r»t i on ) 

Figure  At  Tracking  accuracy.  Target  crossing  distance  10,000  ft. 

Contours  are  spaced  at  Intervals  of  0.02  log  r.m.s. 
error.  The  point  chosen  as  the  centre  for  the  main 
experiment  design  Is  marked  by  a  vertical  cross. 


x^  (vttocity 'V3,  •ccoloration) 

Figure  A2  Tracking  accuracy.  Target  crossing  distance  2,500  ft. 

with  3g  vertical  weave.  Contours  are  spaced  at 


Intervals  of  0.02  log  r.m.s.  error.  The  point 


chosen  as  the  centre  for  the  main  experiment 


design  is  marked  by  a  vertical  cross. 
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Dlaouailon 

A.16.  By  and  large,  results  from  this  series  of  investigations  were 
mutually  consistent.  On  the  face  of  it,  they  appear  to  form  a 
reasonable  basis  for  planning  a  full-scale  experiment.  As  a  centre 
point  for  the  design  of  the  latter,  the  location  P  s  1.8  deg, 

V  =  4.4  deg,  A  =  2.8  deg/seo^.  (x.  =  0.2553,  Xg  =  0.^23)  would 
seem  to  be  satisfactory. 

A. 17.  From  the  evidence  of  this  series,  it  was  concluded  that 
suitable  scales  for  expressing  control  law  parameters  are  given  by: 

Xi 

Xg  =  A/(V  +  0.5/1),  (V  +  2A  =  10) 

where 

P  is  the  value  of  the  position  teim  in  deg. 

V  is  the  value  of  the  velocity  term  in  deg/sec. 

2 

A  is  the  value  of  the  acceleration  term  in  deg/ sec  . 

A. 18.  There  was  slgnificauit  learning  even  during  the  course  of 
Experiment  4,  and  so  it  was  assumed  that  the  design  of  the  main 
experiment  must  allow  for  this  effect.  It  appeared,  also,  that 
ovez^large  blocks  of  tracking  runs  were  to  be  avoided  if  possible. 

On  the  other  hand,  subjects  seemed  to  adapt  to  a  new  control  law 
extremely  quickly,  although  the  possibility  of  long-term  transfer 
of  training  effects  cannot  be  ruled  out.  In  the  short  tens,  however, 

there  would  seem  to  be  little  need  to  provide  lengthy  adapting 

periods  when  switching  to  a  new  law. 

A. 19.  Finally,  it  was  noticed  in  these  experiments  that  settling 
tine  at  the  start  of  a  run  was  greatly  affected  by  control  law. 

If  the  initial  non-soored  period  in  each  run  is  made  very  brief, 
this  could  obviously  have  a  significant  effect  on  final  integrated 
r.n.s.  error.  It  was  decided,  therefore,  that  the  period  allowed 
for  initial  acquisition  and  settling  should  be  increased  to  9  sec. 
in  the  main  experiment,  and  that  acquisition  performance  and  tracking 
accuracy  should  be  assessed  separately. 
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ANNEX  B 

EXPERIMENTAL  DESIGN  AND  ANALYSIS 


Sxperiinental  Design 

B.1  .  In  deaigning  the  main  eiiqperiment  ms  have  considered  the 
following  effects; 

(S)  op6rator"io“operator  -variation 

(A)  order  (or  learning)  affe>3t 

(b)  control  law  parameter a 

(C)  target  course, 

(Not  included  in  the  above  list  is  the  transfer  of  training  effect, 
which  we  have  sought  to  reduce  to  negligible  proportions  by  ignoring 
results  obtained  on  the  first  run  with  each  new  control  law.)  The 
factors  S,  A  and  B  have  been  arranged  in  a  Latin  Square  design,  all 
measurements  taking  place  at  all  levels  of  factor  C.  It  has  been 
asstimed  that  there  is  no  appreciable  interaction  between  factor  A 
(order)  and  factors  S  and  B,  Mo  such  assumption,  however,  has  been 
made  for  the  SB  (subjeots^by-control  law)  interaction.  As  a  result 
it  will,  be  noted  that  the  main  effect  of  factor  A  will  bo  confounded 
with  the  SB  interaction,  although  the  main  effects  of  factors  S  and 
B  (which  are  our  chief  concern  in  this  study)  will  not  be  confounded. 

B.2.  The  control  law  is  Itself  varied  in  two  dimensions  and  the 
proper  placing  of  experimental  points  in  this  plane  requires  some 
oonslderation.  Our  main  aim  is,  of  course,  to  approximate  a 
functional  relationship  between  the  response  (tracking  accuracy 
or  acquisition  time)  and  control  law  parameters  by  means  of  a 
second^order  polynomial.  To  minimize  bias  due  to  neglected  terms 
up  to  and  including  the  fourth  order,  a  so-called  "rotatable"  design 
is  required.  Unlike  the  factorial  design  used  in  Experiment  k  of 
Annex  A,  a  rotatable  de.5lgn  will  generate  information  such  that  the 
response  is  estimated  with  constant  variance  at  all  points  equi¬ 
distant  from  its  origin.  This  is  a  useful  property,  as  little  will 
be  known  in  advance  about  the  orientation  of  any  chosen  design 
relative  to  the  response  surface.  These  points  are  considered  in 
some  detail  in  Refei-ences  B,2  and  E.3» 

B,3.  The  class  of  second-order  rotatable  designs  wo  have  con¬ 
sidered  consist  of  a  ring  of  at  least  five  equi-spaoed  points, 
plus  one  or  more  points  at  the  centre  of  this  circle.  In  Reference 
B.2  it  is  shown  that  when  we  have  reason  to  suspect  the  aoouraoy 
of  our  model,  it  is  safest  to  concentrate  our  points  on  the  olr- 
oumference  of  the  circle  I'ather  than  at  its  centre,  and  to  restrict 
the  scope  of  the  design.  Accordingly,  with  the  number  of  subjeots, 
and  thus  the  number  of  pc.ints,  fixed  at  11,  we  have  placed  eight 
points  on  the  oircxaaf'erenoe  of  the  design  and  three  at  its  centre. 
Working  in  terms  of  the  design  variables  and  Vg,  obtained  from 

the  control  lair  parameters  by  the  auaple  l:Lnear  transformations 
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=  (x^  -  o.255273)/o.1 76091 
Tg  =  (Xg  -  0A82759)/0. 107759 


the  design  matrix  (r)  and  the  derived  matrix  of  Independent 
variables  (V)  are  given  In  Table  B.I.  (The  ssme  design  matr^ 
foxnatlon  Is  given  in  terms  of  the  untransf  ormed  variables  In 
Table  I  of  the  main  text.)  Given  the  measured  response,  , 

at  each  design  point,  the  estimate  b  of  the  vector  Is  given 
by 

b  =  (V'V)“‘'  V'i, 


TABLE  B.I 

EIFEEIMENTAL  DESIGN 


Experimental 

Feint 

Design  Matrix 

Inde] 

Matrix  of 
;)endent  Variables 

^2 

V 

0 

’2 

2 

’1 

2 

^^2 

V1T2 

1 

0 

0 

1 

0 

0 

0 

0 

0 

2 

0 

0 

1 

1  0 

0 

0 

0 

0 

3 

0 

0 

1 

0 

0 

0 

0 

0 

4 

-1 

-1 

1 

-1 

-1 

1 

1 

1 

5 

-1 

1 

1 

-1 

1 

1 

1 

-1 

6 

1 

-1 

1 

1 

-1 

1 

1 

-1 

7 

1 

1 

1 

1 

1 

1 

1 

1 

8 

0 

1 

0 

2 

0 

0 

.9 

0 

1 

0 

2 

0 

0 

10 

0 

1 

0 

0 

2 

0 

11 

0 

-v^ 

1 

0 

A 

0 

2 

0 

Analysis  of  Yarlanoc 

Both  for  tracking  aoouraoy  and  for  acquisition  time  the 
interaotlon  between  control  law  and  target  course  proved  signi¬ 
ficant  well  beyond  the  0.0001  level.  Target  courses  have  therefore 
been  considered  separately,  and  analyses  of  variance  are  shown 
in  Tables  B.II  to  B.VII.  It  will  be  noted  that  the  sums  of  squares 
due  to  control  law  have  been  subdivided  Into  a  posrtion  due  to 
regression  and  a  residual.  In  no  case  did  this  residual  prove 
significantly  different  from  the  subjeot-by-oontrol  law  interaotlon, 
and  so  these  two  terms  have  been  pooled  to  form  a  single  error  term. 
It  Is  worth  noting  from  the  results  that  the  subjeot-by-control  law 
interaotlon  term  reached  the  0.05  level  of  significance  In  only  one 
of  the  six  analyses. 

B,5«  The  fitted  regression  ooeffioients  (quoted  in  a  form  appro¬ 
priate  to  x^  and  rather  than  v^  and  Vg)  are  given  in  Table  II 

of  the  main  text. 
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TABLS  B.II 

ANALYSIS  OF  VARIANCE,  TRACKIN&  ACCURACY 
_ 10,000  ft.  Crossing  Distance 


Source 

SS 

df 

f-— 

MS 

F 

Significance 

S  Subjesjts 

1 .0732 

10 

0.10732 

A  Order 

0.2007 

10 

0.02007 

B'  Control  Law 

(0.1855 

10) 

Regression 

0.1274 

5 

0.02548 

3.19 

p  c  .025 

Residual 

0.0581 

0.00799 

SB 

0.7008 

90) 

1.05 

N.S. 

W  Within  Cells 

0.9015 

0.00745 

TABLE  B.III 


ANALYSIS  OF  VARIANCE,  TEACKIN&  ACCURACY 
2,500  ftc  Crossing  Distance 


Source 

SS 

df 

— 

MS 

F 

Slgnif loanee 

S 

Subjects 

2A173 

10 

0.24173 

A 

Order 

0.0593 

10 

0.00593 

B 

Control  Law 

(0.2111 

10) 

Regression 

0.1739 

5 

0.03478 

4.49 

p  <  .005 

Residual 

0.0372 

5) 

0,00775 

SB 

0,6990 

90) 

1.32 

N.S, 

W 

Within  Cells 

0.7108 

121 

0.00588 

TABLE  B.IV 


ANALYSIS  OF  VARIANCE,  TRACONC  ACCURACY 
5(000  ft.  Crossing  Distance,  Plus  2g  Weare 


Source 

SS 

if 

MS 

F 

Significance 

S 

Subjects 

1.9034 

10 

0.19034 

A 

Order 

0.0681 

10 

O.OO68I 

B 

Control  Law 

(1 .2824. 

10) 

57.96 

Regression 

1  .2766 

5 

0.25533 

p  <  .0001 

Residual 

0.0057 

5) 

0.00441 

SB 

0.4127 

90) 

1.54 

p  <  .05 

W 

Within  Cells 

0.3613 

121 

0,00299 
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TABLE  B.V 


ANALYSIS  OP  VARIANCE,  ACQUISITION  TIME 
_ 10,000  ft.  Croaaing  Distance _ 


Source 

SS 

df 

MS 

P 

Significance 

S 

Subjects 

54.538 

10 

5Jf538 

A 

Order 

3.958 

10 

0.3958 

B 

Control  Law 

(145.270 

10) 

Regression 

140.835 

5 

28.1670 

34.36 

p  <  .0001 

Residual 

4.435 

5) 

0.8197 

SB 

73.435 

90) 

1.24 

N.S. 

W 

Within  Cells 

79.680 

121 

0.6585 

TABLE  B.VI 


ANALYSIS  OP  VARIANCE,  ACQUISITION  TIME 
2,500  ft.  Creasing  Distance 


Source 

SS 

df 

MS 

P 

Significance 

S 

Subjects 

31.347 

10 

3.1347 

A 

Order 

4.716 

10 

0.4716 

B 

Control  Law 

(  16.555 

10) 

Regression 

13.359 

5 

2.6719 

4.20 

p  <  .0001 

Residual 

3.195 

5) 

0.6369 

SB 

57.308 

90) 

1.23 

N.S. 

W 

Within  Cells 

62.586 

121 

0.5172 

TABLE  B.VII 


ANALYSIS  OP  VARIANCE,  ACQUISITION  TIME 
5|000  ft.  Crossing  Distance,  Plus  2g  Weave 


Source 

SS 

df 

MS 

P 

Significance 

S 

Subjects 

97.717 

10 

9.7717 

A 

Order 

19.154 

10 

1 .9154 

B 

Control  Law 

(111.928 

10) 

Regression 

106.179 

5 

21 .2358 

12.69 

p  <  .0001 

Residual 

5.749 

5) 

1 .6732 

SB 

153.204 

90) 

1.29 

N.S. 

W 

Within  Cells 

159.441 

121 

1.3177 
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Canonical  Analysis  of  Fitted  Equations 

B,6,  Some  insight  into  the  nature  of  the  fitted  equations  can 
be  obtained  by  reducing  them  to  their  canonical  form.  In  essence, 
this  consists  of  shifting  the  origin  to  the  centre  of  the  system 
of  contour  curves,  and  rotating  the  co-ordinate  axes  to  correspond 
to  the  principal  axes  of  the  system.  The  quadratic  equations  then 
reduce  to  the  form; 


y  -  ^o  = 


+  B 


22 


where 


y 


0 


is  the  predicted  response  at  the  centre  of 
the  system 


and 


and  Wg  are  the  new  co-ordinate  axes. 


It  is  possible  to  test  whether  differs  significantly  from 

zero,  and  thus  it  is  possible  to  judge  whether  all  the  curvature 
of  the  response  surface  effectively  takes  place  in  one  plane,  so 
that  the  system  can  be  approximated  by  a  surface  containing  a 
stationary  line.  From  Beferenoe  B.3.  Section  10,  the  significance 
of  Bg,  can  be  established  by  referring  the  quantity 


4-  NB 


22 


2(\"''  +1)  c^s 


2  2 


to  tables  of  the  F  distribution  with  degrees  of  freedom  (2,  0) , 

2 

3  is  an  estimate  of  variance  having  ^  degrees  of  freedom, 
derived  in  our  case  from  the  relevant  analysis  of  variance 
table,  with  0  =  95 •  The  other  symbols  are  defined  in  the  reference 
quoted  above.  For  our  design  N  =  11,  \  =  1l/l6,  1  =  8/3  and  c  =  1l/8 
and  so  h  =  2.8235.  Table  B.VIII  lists  the  computed  values  of  B^^ 

and  together  with  the  statistic  defined  above.  As  ^^{2,95)  = 

none  of  the  B^^  coefficients  prove  significant  at  the  0,25  level. 


TABLE  B.VIII 


COEFFICIENTS  OF  CANONICAL  EQUA.TIONS 


Target  Course 
(crossing  distance) 

^11 

®22 

s2 

^22'/-^ 

Tracking  Accuracy 

10,000  ft. 

0.027435 

-0.004624 

0.0003631 

0.166 

2,500  ft. 

0.017573 

-0.009276 

0.0003523 

0.622 

5,000  ft,  +  weave 

0.024611 

-0 .000606 

0.0002002 

0.001 

Acquisition  Time 

10,000  ft. 

0,38892 

-0.03748 

0.03726 

0.107 

2,500  ft. 

0.17511 

-0.01346 

0.02895 

0.177 

5,000  ft.  +  weave 

0.24934 

-0.12547 

0.07605 

0.584 
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B.7«  Thti  equations  of  tht.  iines  forming  the  canoni,cal  axes 
and  Wg  a:re  gixen  by:; 


®12 

c 

Cii  ‘ 

_ t 

“?2 

0 

^2  -  ^2 

where 

M  xa  the  orthogonaj  tians formation  maxri.x 
0  0 

and  and  are  the  orig'j.ns  of  the  aystem. 

of  contours. 


O  0 

The  computed  elements  of  M  and  the  vod-uea  of  Vj  and  Vg  are  given 

in  Table  Bo. IX,  Now  in  a  sense  it  is  true  than  the  principal  axes 
are  the  lines  along  which  there  is  respectively  least  euid  most 
variation  of  response.  But  bj’- ."variation"  is  meant  variation  per 
unit  distance,  and  it  must  be  borne  in  mxnd  that  the  scales  v,  and 

’  I 

Vg  are  entirely  arbitrary,  with  units  which  are  not;  co.mpai‘abl8 .  A 

simple  change  of  scale  (from  v.j  and  v^  to  x.^  and  x^,  for  instance) 

would  bring  about  a  different  orientation  of  the  principal  axes.  The 
canonical  axe.s  found  he.re  and  illustrated  in  Flgr^es  t  to  7  (main 
text)  and  A.'l  to  A. 2  are  those  obtained  using  the  experimental  scales 
.and,  v.,,  and  the  lind,tatio.ns  in  their  interpretation  m.ust  be  borne 

in  mindo 


TABLE  B,IX 


ELEMENTS  OF  ORTHOGONAL  TRANSFORMATION 
MATRICES  AND  ORIGINS  OP  CANONICAL  SYSTEMS 


Target  Cour,se 
(crossing  distance) 

A 

1  1 

®12 

“21 

®22 

0 

0 

^2 

Tracking  Accuracy 
10,000  fto 

2,500  ft, 

5,000  fto  +  weave 

0,9037 

0,9867 

0,9900 

-0,4282 

0,162.5 

0.1408 

0.4282 

0,1625 

0^1408 

0,9037 

-0,9867 

-0,9900 

0,3497 

-0,5985 

-0,1016 

0.4699 
-  0.4953 

-10,8987 

.Acquisition  Tin.\e 
10,000  ft„ 

2..  500  ft, 

5..  000  ft,  +  weave 

0.9861 

0,9895 

0,886o 

0,1662 

-0,1446 

0,4637 

0,1662 

0,1446 

0,4637 

-0,9861 

0,989.8 

-0,8660 

-1,7820 

0,6748 

-1,5587 

4.9769 

7.0287 
-  0.0956 

Confidence  Interyal,s 

Bo8,  The  width  of  the  1  -  a  confidence  interval  for  the  predicted 
response  at  a  given  poi,Dt  (v^°,  v„°)  will  be  given  by; 

i  s^(v'")'(y;i)'V  }  -  ^^1-72'' 
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where  ^ 

s  is  an  estiinate  of  variance  baaed  on  0 
degrees  of  freedom 

and  ^1_a/2  Student's  t  at  the  appropriate 
^  percentage  point. 

For  the  design  used,  k  =  5/8  at  any  location  on  the  outer  clrole 
of  points.  The  width  of  the  50jS  and  90jS  confidence  intervals  for 
y  at  such  a  location  are  given  in  Table  III  of  the  main  text. 

Variance  Due  to  Operators 

B.9.  It  is  important  to  know  vdiether  there  is  the  same  spread 
of  operator  tracking  ability  (measured  in  log  units)  for  all  the 
target  courses  in  this  experiment.  In  fact,  the  hypothesis  we 
wish  to  test  is  that  the  variance  due  to  operators  is  the  same 
for  each  target,  and  that  all  oovariances  are  equal.  The  matrix 
of  observed  variances  and  covariances  of  tracking  score  totals 
is  given  in  Table  B.X.  Applying  Box's  test  of  the  above 

hypothesis^®*^ \  we  find  F(if,2400)  =  1.97,  which  fails  to  attain 
significance  at  the  0.03  level. 


TABLE  B.X 

COVARIANCE  HATRIX 
OPERATOR  TRACKING  ERRORS 


Target  Course 
(crossing  distance) 

(i) 

(li) 

(iii) 

(i) 

10,000  ft. 

2.3811 

3.1892 

2,2912 

(ii) 

2,500  ft. 

3.1892 

5.3180 

4.0737 

(iii) 

5,000  ft.  +  weave 

2,2912 

4.0737 

4.1874 

B.10.  In  the  statistical  model  for  the  analysis  of  variance  based 
on  the  data  for  all  target  courses,  the  expeoted  value  for  the  mean 
square  for  subjects  is  given  by: 

E(MS^)  =  0^^  +  66 

and  that  for  the  within  cells  mean  square  by: 

E(«S,)  =  0,2 

where  g 

is  the  variance  due  to  error 

0 

2 

and  is  the  variance  due  to  operator  differenoes 

^  in  the  population. 

Our  estimate  of  the  variance  due  to  operator  differenoes  is  thus: 
s^  =  (MS^  -  MS^)/66 

=  (0.469311  -  0.005437 )/66 
=  0.007028 
=  0.08383 


and  so  s 
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and  a  range  +  1 .6^+49  a  gives  0.2768,  equivalent  to  a  factor  of 

1.9.  Using  the  approximate  method  of  Satterthwaite^®*^\  the  50^2 
oonfidenoe  inteiral  is  given  by  0.0740 < o < 0.1001  and  the  90^ 
interval  by  0.0579  <  ^  <0.1320. 

Relationships  Between  Aocuracv  and  Other  Variables 

B.11.  The  linear  regression  equation  describing  the  relationship 
between  individual  mean  log  r.m.s,  tracking  errors  (^)  and  mean 
acquisition  time  (x^)  is: 

y  =  0.35240  +  0.0023Qx^ 

It  win  be  noted  that  the  relationship  is  positive,  an  increase 
in  tracking  error  being  associated  with  an  increase  in  acquisition 
time.  From  the  analysis  of  variance  (Table  B.Xl)  it  will  be  seen 
that  the  relationship  is  statistically  significant,  and  that  the 
product-moment  correlation  coefficient  between  the  two  indices  is 
0.878. 


T|LBLE  B.XI 

RECRESSION  ANALYSIS 
TRACKINS  ACCURACY  vs  ACQUISITION  TIME 


Source 

SS 

df 

MS 

F 

Significance 

Regression 

239.006 

1 

239.006 

30.41 

p  <  .0001 

Residual 

70.742 

9 

Total 

309.748 

10 

B.12.  For  tracking  accuracy  and  TSC  score  (x,  )  the  regression 
equation  is:  ^ 


^  =  0.65723  +  0.0058x^ 

This  represents  an  expected  increase  in  r.m.s.  tracking  error  of 
approximately  1  ,yfo  per  point  increase  in  TSC.  This  compares  with  a 
figure  of  592  found  in  the  experiments  reported  in  Reference  B,5. 
From  the  analysis  of  variance  shown  in  Table  B.XII  it  will  be  seen 
that  the  relationship  obtained  here  fails  to  attain  statistical 
significance , 


TABLE  B.XII 

REGRESSION  ANALYSIS 
TRACKING  ACCURACY  vs  TSG  SCORE 


Source 

SS 

df 

MS 

F 

Significanoa 

Regression 

31 .059 

1 

31 .059 

1.00 

N.S, 

Residual 

278.689 

9 

Total 

309.748 

10 
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